The system disclosed herein relates to receiving, processing and storing data from many sources, representing the most “correct” summary of facts and opinions from the data, including being able to re-compute this in real-time, and then using the results to respond to queries. As an example, when a user inputs a query to a web-based system, mobile phone, or vehicle navigation system searching for a “child friendly Chinese restaurant in Greenwich Village that has valet parking”, the system can very quickly respond with a list of restaurants matching, for example, the attributes: {“kid_friendly”:true, “category”:“Restaurant>Chinese”, “valet_parking”:true, “neighborhood”:“Greenwich Village”}. A mobile phone may then provide a button to call each restaurant. The information describing each restaurant may be spread across many websites, sourced from many data stores, and provided directly by users of the system.
A problem in the art is that all web pages, references, and data about all known businesses in the United States stored in any data store can be so large as to not be understandable and query-able in real-time. Updating and maintaining such a large amount of information can be difficult. For example, information describing businesses in the United States has more than billions of rows of input data, tens of billions of facts, and tens of terabytes of web content.
At the same time, new information is continuously becoming available and it is desirable to include such information in the production of query results. As an example, the system may learn that a restaurant no longer offers valet parking, that the restaurant disallows children, or that the restaurant's phone number has been disconnected.
Accordingly, it is desirable to be able to update a system that produces search results both on an ongoing basis (e.g., to account for newly written reviews) as well as on a whole-sale basis (e.g., to reevaluate the entire data and use information contained within that may have been previously unusable).